Performance: Reduce HashMap/Collection Allocation Overhead in Gateway Path

Motivation

JFR profiling of the baseline (main) under high-concurrency gateway workloads revealed that HashMap-related allocations (HashMap$Node, HashMap, HashMap$ValueIterator) and HTTP header collections (DefaultHeaders$HeaderEntry, HttpHeader) are responsible for a significant share of total object allocation churn.

Baseline JFR allocation profile (c128 Read HTTP/1, ObjectAllocationSample, 10-min recording):

Root causes:

1. HashMap<>() default initial capacity (16) forces 1-2 resize+rehash cycles for typical gateway responses with 20-30 headers, creating throwaway HashMap$Node[] arrays and re-hashed HashMap$Node entries
2. StoreResponse constructor converts HttpHeaders to Map via HttpUtils.asMap() on every response, allocating a throwaway HashMap$ValueIterator and rebuilding all HashMap$Node entries
3. HttpHeaders in RxGatewayStoreModel.getHttpRequestHeaders() is undersized, causing internal HashMap resize
4. Redundant toLowerCase() calls on header keys that are already normalized

Changes

1. Right-sized HashMap initial capacity: HashMap<>(32) instead of HashMap<>() in RxDocumentServiceRequest, and mapCapacityForSize() helper in HttpUtils to avoid rehashing
2. Eliminate HashMap to HttpHeaders to HashMap round-trip: StoreResponse now accepts HttpHeaders directly, removing intermediate asMap() conversion that created throwaway HashMap$ValueIterator and HashMap$Node arrays
3. Pre-sized HttpHeaders in RxGatewayStoreModel: sized to defaultHeaders.size() + headers.size() to avoid internal HashMap resize
4. Remove redundant toLowerCase() calls: HttpHeaders.set() already normalizes keys; callers no longer double-normalize creating extra String objects

Benchmark Results

Test matrix: 1 tenant x {c1, c8, c16, c32, c128} concurrency x {Read, Write} x {HTTP/1, HTTP/2} x 3 rounds each, GATEWAY mode, 10 min/run.

Throughput Summary (ops/s, 3-round average +/- stddev)

Variance Analysis

The apparent -4% to -6% deltas at mid-concurrency (c16/c32) are not SDK regressions they are caused by server-side transit time variability between rounds.

A dedicated 6-round reproducibility study (1t-c32-ReadThroughput-http1) with request-level metrics enabled confirms this:

The request-level breakdown shows the variance lives entirely in transitTime (server round-trip), not in SDK-side processing:

hashmap-alloc has 4x lower CV (1.8% vs 7.2%).

Root cause of transit time variance (confirmed via TCP socket analysis): The bimodal pattern is caused by Azure Traffic Manager (ATM) routing the regional Cosmos endpoint (benchmark-cosmos-lx1-westus2.documents.azure.com) to different frontend nodes on each JVM restart. Confirmed by running ss -i -t -n state established '( dport = :443 )' mid-run across 6 rounds all 32 connections land on the same IP within each round, and that IP alternates perfectly with throughput: 20.9.156.133 (fe2, slow ~19.3K ops/s) vs 20.42.170.147 (fe7, fast ~20.2K ops/s). The ATM CNAME record has TTL=20s, so each JVM restart resolves to whichever frontend ATM selects at that moment. This is infrastructure variability unrelated to SDK code.

GC Comparison (c128 Read HTTP/1, r1)

GC behavior is identical between branches. At single-tenant scale with an 8 GB heap, the allocation reduction does not materially change GC frequency or pause time. The benefit is reduced unnecessary work (fewer resize/rehash cycles, fewer throwaway iterators) which would compound at higher tenant density.

JFR Allocation Comparison All Configs

ObjectAllocationSample comparison for aggregate allocation share of all 9 targeted classes.

Note on HashMap$ValueIterator: This PR eliminates the response-side HttpUtils.asMap() iterator. A separate HashMap$ValueIterator still exists on the request-sending side (ReactorNettyClient.bodySendDelegate) this is expected and not targeted by this PR.

Note on JFR sampling noise: Individual per-config percentages can swing +/-5pp between runs. The consistently observable patterns are:

1. HashMap$ValueIterator is eliminated in most configs (the asMap() round-trip is removed)
2. At high concurrency (c128), targeted allocation share drops consistently (Read/HTTP1: 20.4%17.4%, Write/HTTP1: 16.5%10.1%)

Detailed breakdown for c128 Read HTTP/1 (highest pressure, most stable JFR signal):

Summary Chart

30-Tenant Benchmark Results

Test matrix: 30 tenants x {c3, c5} concurrency-per-tenant x {Read, Write} x {HTTP/1, HTTP/2}, GATEWAY mode, single cycle (~10 min steady-state). Metrics reported from tenant 0 (representative).

7 of 8 configs are within 3% throughput. The 30t-c3-Write/HTTP2 config shows -5.6% within ATM-induced between-run variance (runs separated by ~4 hours, ATM may route to different frontend). The 30t-c5-Write/HTTP1 config shows +4.8% throughput with p99 reduced from 9.29ms to 7.72ms (-17%).

Conclusion

• Throughput: neutral overall (3% across all configs), within measurement noise
• Variance: apparent regressions at c16/c32 are server-side ATM routing variability confirmed via TCP socket inspection (ss -i); hashmap-alloc has 4x lower throughput CV (1.8% vs 7.2%)
• GC: identical (817 vs 813 pauses, same mean/p99)
• Allocation efficiency: HashMap$ValueIterator eliminated; HashMap$Node -23%, DefaultHeaders$HeaderEntry -35% at c128
• 30-tenant: neutral across 8 configs (7/8 within 3%); 30t-c5-Write/HTTP1 shows +4.8% with -17% p99
• The changes remove unnecessary allocation overhead without regression. The benefit compounds at higher tenant density where allocation pressure and GC become bottlenecks.